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ABSTRACT 


One of the most crucial problems in theoretical and applied statistics ıs to 
determine the precision of the estimates produced by different statistical estimators. 
This problem is greatly increased when the population parametric characteristics are 
not known. Parallel to this problem is that of deciding how large (or small) the sample 
population must be in order to obtain a desired precision within certain range. 

There are several non-parametric methods to approach the first problem. The - 
BOOTSTRAP Method (Efron, 1979) is one ofthese approaches and the one of interest 
in this thesis. With this method, one could improve the precision of the estimates and 
gain information about the distributional characteristics of statistical estimators. The 
bootstrap method has been amply compared with other methods; the results show that 
the bootstrap method often produces more precise estimates (i.e. with smaller mean 
squared error) than competitors such. as the JACKNIFE, SECTIONING and 
CROSS-VALIDATION. However, the results that have been obtained are based on 
large sample sizes and large numbers of "bootstrap" replications. 

This thesis analyzes the behavior of the BOOTSTRAP method when the number 
of bootstrap replications 1s small. It tries to identify any tradeoffs between sample size 
and the number of bootstrap replications required to attain a desired precision in the 
estimates produced in several particular situations. One of the goals is to produce 
graphical displays that will indicate to the experimental statistician the price that must 
be paid in the precision of the estimates, obtained with the bootstrap method, when 


sample size is small, and the number of bootstrap replications to use in this situation. 
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I. INTRODUCTION 


A. BACKGROUND 

One of the most common problem in applied statistics is the estimation of an 
unknown parameter 0. Once the statistician has decided on the model having one or 
more parameters to be estimated and has selected the estimator (1.e., m.l.e., least-square 
estimator, etc.) that will be used to obtain the estimates, the second problem that he or 
she faces is how to estimate the accuracy of these estimates. There are several ways of 
measuring the accuracy or the error of statistical estimators. In this thesis, the measure 
of statistical error will be defined to be the mean squared error (MSE) of the 
estimators; i.e. the variance plus the bias-squared of 0^" (where OP represents the 
estimator of the parameter 9. In Appendix A the reader will find a list of special 


notations used in this thesis) : 
MSE(0^) - E[((0^ - 60)?^|] 2 Var(0^) + [BIAS(0*)]* ΠΠ 


When the practitioner is dealing with samples obtained from populations for 
which the distributional characteristics are known, classical statistical theory provides 
an answer to the second problem that the statistician faces. This is true since, at least 
in theory, the variance and the bias of most statistical estimators can be calculated 
analytically. However, the difficulty of analytically deriving the MSE of some statistical 
estimator increases as the mathematical definition of the estimator becomes more 
complicated. When this is the case or when the practitioner does not actually know the 
probability distribution, say F, from which the sample was obtained, then the MSE of 
the estimators must be estimated. 

There are several non-parametric methods for estimating the bias and the 
variance of an estimator of interest. The most common ones are the Quenoille-Tukey 
JACKNIFE method, CROSS-VALIDATION, and SECTIONING; the Jacknife being 
the most commonly used of the three approaches. Efron and Gong [Ref. 1] and Miller 
[Ref. 2] provide an excellent exposition of the first two methods and Lewis gives a good 


introduction and analysis of the later (See [Ref. 3] ). 


and then X ug F^. Then the task is to estimate the distribution of O(F) by the 
distribution of 0 (F^), where 0 (F^) denotes the value of the parameter of interest 
based on the bootstrap mechanism. This mechanism proceeds as follows : keeping F 2 
fixed, draw a bootstrap sample and calculate 0 (F h), do this a large number B of times 
obtaining 0" (F^), 0 (F^), om 9 (ΕΛ). The resultant (sample) distribution οθ΄ is 
called the bootstrap distribution F h% Once F”* is obtained, then any specific feature 
of this distribution, such as expected value of θ᾽ Ex(0 ) or the variance of 0“ ; 
Маг» (θ΄ ), could be obtained. (In this thesis, notation like "E« ", “Var ^, "S 2 x ο ΄ 
, etc., indicates calculations relating to the conditional bootstrap distribution of x“ , with 
the vector of random variates X and hence F^ , fixed.? ). Theoretically, then, the 
bootstrap idea could be used to estimate the expected value, the variance, and the 
mean squared error of any estimator, given a sample that comes from an unknown 
probability distribution F. 

As mentioned earlier, Efron (See [Ref. 4] ) has shown that this method is often 
more precise than other non-parametric methods for assessing statistical accuracy. 
However, the experimentation done in the past using this method relied on a /arge 
number B of bootstrap replications; i.e, a large sample on 0 . In some cases, it can be 
shown (see Chapter 2, for the case of Var«(0 )) that as B =, the variance of θ᾽ 
based on FP is equal to the variance of the estimator O based on F . But, how large 
must B be in order to obtain estimates that are accurate or to obtain estimators with a 
small MSE is a question to be answered. Also, what is the tradeoff between the 
sample size n and the number B of bootstrap replications ? 

The purpose of this thesis is then twofold : first, to analyze the bootstrap 
performance as the number B of replications increases, starting from a small B. The 
second, also of great interest, is to study the relationship between the sample size n and 
the number B in the estimation of the MSE of the estimator using the bootstrap 


mechanism. 


C. ORGANIZATION 
There are several methods of dertermining the bootstrap distribution of an 


estimator ϱ(Ε ^) two of which will be analyzed in this thesis.” The first is by direct 


2As it will be shown in the next cha ter, this is critical feature of the 
BOOTS TRAP method: the vector of random ana X and F m must be fixed through 
cess 


3A third method involves making Taylor series expansion to obtain the 
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theoretical calculations (this is usually the most difficult approach). The second relies 
on Monte Carlo approximations to the bootstrap distribution: repeated realizations of 
Me are generated by taking random samples of size n from FP , say Е И ; s Г, 
х“ апа ће histogram of the corresponding values 0^ ,(Fh) 0 (Fh) ER πο is 
constructed as an approximation to the actual bootstrap distribution (See [Ref. 1: 
Section 2] ). These two methods are of interest in the second chapter. In the last 
section of Chapter Two, the different statistical experiments conducted for this thesis 
are explained in detail. In Chapter Three, the results from these experiments are 
presented and analyzed, and the problem of using the bootstrap approach in linear 
regression problems is also discussed. Conclusions are presented in the last chapter. 
There, one of the points of interest is to discuss the main disadvantage of the bootstrap 
methodology : the computer time required to implement this method when Monte 
Carlo simulation is used. In Appendix B, the FORTRAN software that was designed 
to run the experiments discussed in this thesis will be explained and the code is listed. 
This computer program. is user friendly and can be used to estimate the bootstrap 
distribution of eight different estimators. Finally in Appendix C, the reader can see 
some tables that give a good idea about how large (or small) B and n can be in order 


to obtain a desired precision on the estimates of parameters of given populations F. 


approximate mean and variance of the bootstrap distribution F . See Ref.4, Section 5. 
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l. Direct Analytical Calculations 
An attempt is now made to calculate some parameters of interest of the 
distribution of X ;. Assuming the conditions shown in expressions (2.1) and (2.2), the 


expected value of x , given X, could be calculated as follows : 
* * * 
EX 4) = EX IX = x) = X RO A (2.3) 
where j = 1, 2,..., n. From (2.2), this is equal to : 
EX; ) = = (x; / n ) = X - 1Ξ 1522. ee о А (2.4) 


which is the sample mean of the original sample X. Then from (2.4), the unconditional 


* . 
expected value of X ; is: 
E (X^) - EIE«X”, | X)) = E(X) = p, j21,2,.., n. (2.5) 


Thus, the unconditional expectation of X ; is equal to the mean of the population 


from which the original sample was eu (Note, from this point on all summation 
signs go from 1 to n, unless otherwise specified, and Ex , Varx , etc., are conditional, 
glve X .) 

Likewise, the unconditional variance of x* could be derived from the 


conditional variance of X : 
* * * 2 
Var«(X |) = E«[(X ce E(X 4 Х = х)) |. (2.6) 
Using (2.5) this expression is equivalent to : 


Var(X 1) 2 EX ;- X)*| X] (2.7) 
В 
Y; КА. n) - X? 
= X2 
=), (X; -X)®/n 
By definition ofthe sample variance, саз chen 


Var«(X",) = (n-I)/n S?, (2.8) 
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Now, unconditionally 


E(Var«(X )) + Var[E«(X )] (2.9) 
= 2 S x 
- E[»; (X5 / n) - X*] * Var(X) 
E [(n-1)/n 5] = вс /n 
(n-1)/n E(S*,) + 0°, /n 
2 2 

πη, + sn 

2 
Ον 
Therefore, the variance Шр опдела!) of X, ; is the same as the variance of 


Var (X^) 


Х;. Тһе covariance between x”. and X has a - important impact on the 
Eas methodology, Bicis when the bootstrap distribution of θ᾽ (FP) 15 
approximated by Monte Carlo simulatıon (see next шш 


Conditionally (given X), the covariance between X. ; and X j ÍS as follows : 
Cova(X* X") E E |) (κ᾿ ! E«(X 3) ] | (2.10) 
From (2.5), this is 
Cova(X" X”; ) - EXT; - X) x^ 2508 (2.11) 


Бах", х", ¡)- x2 


Now conditionally, given X = x, oe joint distribution of x”; 24 p E uniform over the 


points (re νο x (KK go "x τ and this ımplies that x“. vs p = (ххх) with 
probability 1/n?. Then 
* * А : 
Ex(X {Х = 0 (x; x; ) / n2 ES (2.12) 
= (1/n) È; x)? = xš. 


Finally, the conditional covariance nn > qe ; and х", L 


Cov«(X X ) = X2.-X2 = 0. (2.13) 
Now, to derive the unconditional covariance between X. ; and X , It will be convenient 


to use the result obtained in equation (2.13). To use (2. 13), it E. be shown that the 
following equality holds: 
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Cov(X*;,X*;)= E[Cov=(X* X] + Cov[Es(X*;), E«(X JJ. | (2.14) 


To show this, notice that the conditional covariance can be defined as 


Cov(X,Y|Z) = Ey AY - E(X|Z)ECY|Z))|Z] (2:15) 
= Evy yizy (XYIZ) - [EXIZE(YIZ) . 
Then 
E,[Cov(X,YIZ)] = EdE(x yz ҮҮ!) - {E(XIZ)ECY|Z)}] (2.16) 
= E [Eg yjz\(XYIZ)I - {Ε2|Ε(ΧΙΖ)]ἘΖ[Ε(ΥΙΖ)]} - 
- E[E(XIZ)E(YIZ)] - CEZ[E(XIZ)]EZ[ECYIZ)I) 
= Cov(X,Y) - Cov[E(X|Z),E(Y|Z)]. 
Therefore, 
Cov(X,Y) = E,[Cov(X, Y|Z)] * Cov[E(X|Z),E(Y]Z)]. (2.17) 


With this in mind, the unconditional covariance could finally be computed by using 
(2.15). Now, the portion inside the brackets of the first term of the right hand side of 
equation (2.14) was shown in (2.13) to be equal to zero. Then, using expression (2.5), 


equation (2.14) reduces to 
A NN ==. _ S. ο 
Cov(X ;,X j) Ze ο ο ασ ο (2.18) 
and from (2.18), the correlation coeflicient is given by 


P SX) = In = PX; = Xj (2.19) 

Comparing equations (2.13) and (2.18) it could then be stated that the 
bootstrap samples are (conditionally) independent as long as X is held fixed. 

It is possible now to derive the distributional characteristics of some statistical 
estimators based on the distribution of XM In doing this, it is assumed that the 
original sample X is fixed and these derivations are conditional. For example, the 
expected value and the variance of X (the bootstraped sample mean) are obtained as 


follows: using equation (2.5) 
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ЕХ у= х; (2.20) 


so unconditionally, the expected value ofthe bootstrap sample mean is 


* 


ЕХ) = EX) = H, . (2.21) 
The conditional variance of the bootstrap sample mean is 


Vars(X ) 7 (1/n9)Var« [ Y; (X. 3] (2.22) 
= (1/12) (Y; Vars(X" ^ (n(n-1 y2)Cov«(X "X" )] | 
From equation (2.13), the conditional variance is then 
Vars(X ) 7 (1/n)y;Var«(X |) ] (2123) 
- (1/n?) [n Var«(X !)] . 
Using equation (2.8), finally 
Np amne sce (2.24) 


; τε 2*. . 
With this expression, the unconditional variance of X is given by 


Var(X ) = E[Vare(X )] + Var[Ex(X )]. (2.25) 


From equation (2.5), and (2.20) 
Var(X )= El(n-1)/n? S?,] + Var(X) 
= (n-1)/n* ο. = DE /n 
= (2n-1)/n Var(X) 
As mentioned earlier, equation (2.24) is the one of interest when one wants to apply 


the bootstrap mechanism to obtain the variance of X'. Notice that as n — co 
Var«(X ) — Var(X) (2.26) 


strongly (strong law of large numbers), but this is not the case for the unconditional 


° = k 
variance of X , where as n > %, 


Var (X) > 2Var(X). (207) 


It is now possible to define an estimator for the MSE of the mean of a 


population based on X: 


MSE«X ) 2 Var«(X ) * [EX  - E X )f. (2.28) 


| 


Var«(X ) ^ [Bias«(X )]° 


In the same manner, the MSE of any estimator could be derived. However, it 


is easy to see that as the mathematical definition of the estimator gets more 
complicated, this procedure can become very tedious. This is why it is desired to 
estimate the bootstrap distribution of the estimator by simulation rather than 
analytically. 
2. Monte Carlo Simulation 
The algorithm presented in Chapter II, Section A, could be expanded to allow 
Monte Carlo simulation to approximate the bootstrap distribution of 0 (F^). As before 
(See Eiron [Ref СВЙ: 
(1) given that the realization of the random vector X has been observed, say X; 
= x; for ı= 1, 2...,n; 
(2) construct the sample probability distribution F" , by giving a mass 1/n at each 
Point oe no OMS 
(3) keeping x; (and thus, F^ ) fixed, draw with replacement a random sample of 
size n from FP , and call this a bootstrap sample; 
(4) from this random sample, compute the bootstrap replication, 0. (F^); rey 
compute the value of the desire statistic based on the sample from F^ . Then, 
(5) do steps (3) and (4) a "large" number B of times. In this way one obtains 
independent bootstrap replications of 0 (F^), say ga ab), μη). 0" (FR) 


(6) now, approximate the variance of 0 (Fh) by the sample variance 
ж % — 
Var«^ [Ü (F^)] 2 Y; [0 (F^) - 6 (F^? /(B-1), (2.29) 
where i7 1, 2,..., B, and 
=>»; ж 
0 (F) 2 50 (F5/B. (2.30) 
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The MSE of 0 (F^) inay be estimated by 


MSE,4^(0 (F^) — Vars^[0 (F^)] 4 [BIAS."(0 (FJ. (2.31) 


It will be seen in Chapter Three that as B and n get large MSE4'(0 (F^)) approaches 
zero. A problem in using the bootstrap is the choice of B, and we consider this in 
Chapter Three. 

This bootstrap simulation procedure was carried out to study the effect of 
possible choices of B, in terms of the estimated MSE of several estimators. The reader 
will see, in the next chapter, that the choice of B should depend on the sample size n, 
the specific estimator under consideration and the structure of the population from 
which the sample was obtained. 

a. The Statistical Experiment 

In this thesis, various experiments were conducted to study the problem 
of selecting B. The main idea behind these experiments was to select some well known 
probability distributions and some parametric estimators for which the distributional 
characteristics are well known. Then the MSE of these estimators could be determined 
theoretically. Therefore, one could compare this true MSE with the estimated MSE of 
the estimators obtained using the bootstrap mechanism. 

The critical part of the experiment was to design an effective computer 
code to perform the Monte Carlo simulation. The FORTRAN program developed to 
carry out the simulation reported here is listed in Appendix B. This program was used 
to analyze the performance of eight different estimators based on the bootstrap 
methodology. These were the sample mean, variance (three different estimators), 
coefficient of correlation, coefficient of variation, the five-percent trimmed mean, and 
the median. 

The simulation runs as follows (See Appendix B): 

(1) mn random variates, for up to 8 values of n, are first generated representing a 
random sample from a population F. ( In the simulation a total of N random 
variables are first generated, then sectioned into samples of sizes n; where 1— 
1:2 δι) 

(2) For each subsample of size n , a bootstrap function is called to generate a 


bootstrap sample from the original sample. Then, the estimator function is 


18 


called to produce a desired estimate. This step is repeated until B bootstrap 
samples from the original sample are obtained. 

(3) After the B estimates have been obtained, the statistics function is called to 
calculate the mean of these estimates, this number is one of the 0 (F^) 

(4) In order to improve the precision of the simulation process, steps (2) and (3) 
are replicated M times. Then, the process will produce a total of (N X M)/ n 
estimates. From these estimates, a box-plot is constructed and estimates, 
including MSE, are calculated. 

In the next chapter some of the results obtained from this simulation 


process are analyzed. 


I9 


IIl. APPLICATION OF THE BOOTSTRAP METHOD : SOME RESULTS 


A. THE MEAN, VARIANCE AND THE COEFFICIENT OF VARIATION OF 
EXPONENTIAL RANDOM VARIATES i 


The first experiment conducted was intended to analyze the bootstrap mechanism 
in estimating the MSE of the estimators for the mean, variance and coefficient of 
variation of a sample coming from a population of exponential random variates with 


parameter A = l. The population coefficient of variation is defined as: 
СУ(Х) = ву/н, (3.1) 


In the Exponential(1) case, the mean, variance and the coefficient of variation have the 
same value of 1. With this first fact in mind, the MSE of sample mean, as an example, 
Is defined using (2.21) and (2.28) as: 


МЅЕ(Х") = Уақ") * [EX  - n]? . 80) 
Conditionally, from (2.26), an estimate of (3.2) ıs: 
КОШ О [(n-a)/n? S,?] + [Es(X" - 1)? . (3.3) 


In the same manner, the MSE for the variance and coefficient of variation could be 
estimated. These estimates were obtained using the algorithm described in the 
preceding section. The sample sizes for this experiment were: n = 10, 20, 25, 40, 50, 70 
,100, 140. Each estimator was bootstraped using B=5, 8, 10, 15, 20, 25, 40, 60, 100, 
140, and 500. Figures 3.1, 3.2 and 3.3 below, show how the MSE" for the mean, 
variance and coefficient of variation respectively decreases as both n and B increases. 

A remarkable feature of these plots is that the MSEx" of the bootstrap sample 
variance (Figure 3.2) decreases much faster as the sample size increases than when B 
increases. Observe the big jump in the MSE" when п goes from 10 to 40 relative to 
that of B going from 5 to, say, 40: the jump is much greater in the former. 

Another observation of interest is that the MSE” of the estimates decreases as 


B increases, but beyond a certain threshold very slowly. Indeed, the decrease in MSE." 
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Figure 3.1 MSE«" of Bootstrap Sample Mean: Exp(1). 
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Figure 3.2 MSE" of Bootstrap Sample Variance: Exp(1). 
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Figure 3.3 MSEx" of Bootstrap Coeff. of Variation: Exp(1). 


beyond B 2 50 is barely noticeable. For example, see Figure 3.2, the MSE" of the 
sample variance decreases only by one-thousandth of a unit when B is increased from 
200 to 500 replications. This is also true for the sample mean. However, for the 
coefficient of variation (see Figure 3.3), the MSEs" improved about two percent (.02) 
in the same range for a small sample size (n= 10). These results give an idea of the 
performance of the MSE of the bootstrap estimates of a given estimator. It should also 
suggest to the statistician that once the estimators are performing fairly well (1.e., once 
this threshold has been attained), there is no reason to increase the amount of 
bootstrap replications, since this will not induce a great improvement in the estimates. 
An important point here ıs that when an attempt is made to estimate the sample 
variance using the bootstrap method, the number of bootstrap replications should be 
greater than 100 in order to decrease the MSEx" below 0.6. 

The bootstrap distribution of some of the estimators are shown in Figures 3.4, 
and 3.5 in the form of boxplots and a summary of the distributional statistics. These 
were obtained by using a statistical package, called SMTBIO, developed at NPGS (See 
Appendix B). This package was modified by the author of this thesis in order to obtain 
MSE«". Each boxplot represents the distribution of the bootstrap estimator based on 


the sample size n. 
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Figure 3.4 Bootstrap Dist. of Sample Mean B- 5. 
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Notice, ın Figure 3.4, that the distribution of the bootstrap sample mean 
resembles a Normal, as would be expected by the Central Limit Theorem, with the 
Kurtosis and Skewness oscillating around zero, as n increases. Recall from previous 
section that the standard deviation of X, in the case of Figure 3.4, would be estimated 
by 

STD4,NX) = STD." Jn’, n - NXM/NE(I) 
and STD" is the value shown on the bottom table of this figure. Figure 3.5 shows the 
distribution of the bootstrap sample variance (3.5). Looking at the distribution 
summary, one can say that this distribution is quite similar to that of a scaled 
Gamma(k,p) distribution. Again as n, increases.the Kurtosis and Skewness get closer to 
that of the Gamma, say 6/k, and 2/A/k respectively. Figure B.4 and B.5, Appendix B, 
show the distribution of the same estimators when B = 150. It is easy to see that the 
distributional characteristics for the estimators follow the same patterns as those 
discussed above, where B = 5. The only difference there ıs that, as expected, the 
number of outliers decreases significantly particularly in the case of the sample 


variance. 


B. THE SAMPLE VARIANCE 

This experiment was intended to further study the behavior of the bootstrap 
sample variance for populations with various distributions. The ones discussed in this 
section are the GAMMA(0.5,1), NORMAL(0,1) and LAPLACE(0,1). For this 
experiment, the sample size where n = 5, 10, 20, 25, 30, 50, 60, and B= 5, ὃ, 10, 15, 
20, 25, 30, 35, 40, 50, 100, and 500. In the first two cases, the GAMMA and 
NORMAL distributions, the bootstrap sample variance seems to approximate the 
population variance fairly well when n > 50, where the MSEx" is less than 0.10. 
Figures 3.6, 3.7, and 3.8 show the relation between B, n, and the MSE." of the 
bootstrap sample variance for a Gamma(0.5,1), Normal(0,1), and Laplace(0.1) 
respectively. 

Notice that there is a lot of random variation in the MSE," when B is in the 
range 5 S B «50 for n S 30, and for B« 25 when 30 « n S 60. This random noise 
extends beyond these ranges in the case of the Gamma(0.5,1). Notice that in Figure 
3.6, the lines for the MSEx" of the sample variance when n^ 15, and 20 are above 
that when n7 10 for B « 300. However, when B ^ 500, these lines lie below the one 


corresponding to n= 10. The MSEx" for n= 15, and 20 is actually less than the MSEx° 
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Figure 3.5 Bootstrap Dist. of Sample Variance B= 5. 
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Figure 3.6 MSE« of Bootstrap Sample Variance of a G(0.5,1). 
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Figure 3.7 MSE«" of Bootstrap Sample Variance of a N(0,1). 
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Figure 3.8 MSE" of Bootstrap Sample Variance of a L(0,1). 


for n=10 just after B > 150. In this experiment, it is also true as found for the 
Exponential(1), that MSE" decreases faster as n decreases than when B increases. This 
was also the result in the case of the Laplace(0,1). However (notice the scale of the 
MSE in this case), the MSEx" is quite high. Figure 3.8 shows that for a sample of size 
n < 15, the MSEs"® > 1.0 even when B is as large as 500. It was suspected that 
probably this high MSE” was caused by the mechanism used to generate Laplace 
random varlates. The first method used in this experiment takes the difference of two 
Exponential (1) variates. The second method generates an Exponential(1) and converts 
it to a Negative-Exponential(1) with probability .5 . The histograms, using different 
sample sizes, showed that the first algorithm used to generate Laplace random variates 
was the most effective. In any case, the point here is that for the ranges of n and B 
used in the experiment, the MSEx" of the sample variance for a Laplace(0,1) never 
decreased below 0.2. This was not the case for the other distributions. This suggests 
that the performance of the bootstrap method depends on the distributional properties 


of the population in question as well as the estimator under consideration. 
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C. THREE DIFFERENT ESTIMATORS FOR THE VARIANCE 
In Chapter Two, the expected value and the variance of the bootstrap sample 
mean (X) were derived. In this section, the expected value of the bootstrap sample 


variance, call this 15 2 is calculated. Let 


а 02) (аА (3.4) 
те. 
Note that 
E«(X 8), = (1/1) х2, | (3.5) 
so that 
EY; X; 2) - Y; X^, (3.6) 


τν =, 
Likewise the second moment of X is given by: 


Ex(X"*) = (InP, Xi 2+ DEX |) 151 (3.7) 


As before, RR”) has probability (1/n?) of being any point of the form (xx) so 
from (2.7) 


EX ,X^) - (1/n?) E Y; Xi? + УУ ХХ, | (3.8) 
IDE Y (XiX;)/n° ! 
Now 
ΣΣ хх; = (п-п Ха + УУ ХХИ (3.9) 
= ((n-1)/n È; X) 
= n(n-1)X? 


Then (3.7) can be expressed as 
Εκ 2) = (1/n2)Y; X*, - n(n-))X?] (3.10) 
Finally, using (3.6) and (3.9), the conditional expected value of KS 15 
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Bx(,8°2) » (I/(n-D)E«Y; X j? * п ©) (3.11) 


| 


Π/(3-1|Σ; ΕΗΧ3) - пВ»(Х 2) 


1/(n-1) ( Y; X;* - [(1/n) (Y; X;? * n((n-1))X2]) 
= 1/(n-1) [((n-1)/n)Y;; X;* - (n-1)X? J 
= У; (X;* - Xy / ΤΙ, 


Call this c, ?. Now suppose it is known that X ~N(p,6°)- this restriction is not really 


ll 


required in this context - and it is desired to estimate the variance of X using the 
bootstrap method. As shown in the previous chapter, 


E(X") = py , (3.12) 
so the unconditional expected value of Se is: 
Е(,5*2) = Е.18(,5°41Х) (3.13) 
ο EM 
= ((n-1)/n)o,? 
Then ,S”? 


is a biased estimator for Cae The finite population correction factor might 
- * 
thus be suggested to improve the performance of ;S 2 . Define 


SN AO (3.14) 


an unbiased bootstrap estimator of су . Analyzing expression (2.5) and (3.11), yet 
another estimator for o, * 


. = > > } 
x “an be suggested. Since the value of Ex(X; )=X is known, 
the following estimator for с, also seems reasonable: 


S2 -YQ.-X*n (3.15) 


The third experiment was conducted to compare the performance of these three 
estimators (3.4), (3.14), and (3.15). Figures 3.9, 3.10, and 3.11 show the results of this 
experiment. 


As can be seen, the third estimator, 55 2 in almost all cases outperforms the 
other two for all different sample sizes tried in this experiment. Even the second 


estimator (3.14) performs almost as good as κος when n > 50. When n 2 50, the 
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Figure 3.9 MSE" of the Sample Variance of a N(0,1). 
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Figure 3.10 MSEx" of the 2nd Variance Estimator of a N(0,1). 
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Figure 3.11 MSEx" of the 3rd Variance Estimator of a N(0,1). 


difference between these three different estimators is barely noticeable. However, for 
very small samples, n < 20, E is definitly a better estimator for o* than TONS 
Efron [Ref. 1] has suggested the use of E as the bootstrap estimator of the sample 
varıance. As the plots suggest, ıt could be now recommended the use of Es and 
even s (for larger samples, n > 50) rather than se to estimate the sample 
variance. Note that as n— 00, δα is the same as S . (Note: these two estimators 
(3.14) and (3.15) are called VARIA2 and VARIA3 respectively in the FORTRAN 
code, listed in Appendix A). 

THE CENTER OF A DISTRIBUTION: COMPARISON OF THE MEAN, 

MEDIAN AND TRIMMED MEAN 

The sample mean is the most used estimator for the center of a distribution. 
However, two other estimators are also used, specially for symmetric distributions: the 
median and the 5% trimmed mean. There have been many comparisons of the 
asymptotic performance of these three estimators. Lehman [Ref. 8] has calculated the 
asymptotic values of these estimators in case when the sample is from a Normal(0,1) or 


a Laplace(0,1) population. These calculations are summarized in Table 1 below. 


Sil 


TABLE I 
ASYMPTOTIC ο. OF THE MEAN, MEDIAN 


% 


AND RIMMED MEAN 


ESBEMATFORT 


Probability Р 
Distribution Medıan 5% Trimmed Mean 


а 1.57 /п 1.01/n 
Laplace(0,1 1.00/n 1.65/n 





These values, among other things, show that for the case of sample coming from a 
Normal(0,1), the mean has less asymptotic variance than the other estimators. 
However, if the data comes from a population with heavy tails, like the Laplace, the 
median is a better estimator asymptotically (having less variance). The 5% trimmed 
mean is a compromise between the other two: it should used when the practitioner 
does not know the nature of the tails of the population. 

A fourth experiment was conducted to see if these observations hold when the 
corresponding bootstrap estimators are used. In this experiment, the MSE of of the 
bootstrap estimators were compared with the asymptotic MSE for the usual estimators 
as B increases. The asymptotic MSE (call it MSE,) of the three estimators could be 
estimated by adding the asymptotic variance, as defined in Table 1, plus the 
bias-squared. The MSE, was compared with the MSE" of the bootstrap 
estimators, for several sample sizes, as B increases. 

Figures 3.12, 3.13, and 3.14 summarize the results of this comparison for the case 
of a Normal(0,1) population. Figures 3.15, 3.16, and 3.17 show the results for a 
Laplace(0,1) population. 

In these figures, the solid horizontal lines represent the values of the asymptotic 
MSE of the usual estimators. For example, in Figure 3.12 the estimated asymptotic 
MSE of the sample mean for a sample of size n- 5 is approximately 1/5.0 + 
(BIAS)*~.20. The dotted line represents the estimated MSE of the bootstraped 
estimators as B increases. 

In summary, for the Normal(0,1) population, the bootstraped sample mean and 


the 5% trimmed mean have less error, asymptotically; they are estimating the center of 
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the distribution with much better precision than the bootstrap sample median. 
Comparing Figures 3.12 and 3.13, it looks obvious that for sample sizes n 60 the 
bootstraped sample mean shows much smaller MSE than the bootstraped sample 
median. When the sample size 1s n 60 there is no distinguishable difference between 
the estimated MSE's of these two estimators. Notice that the bootstraped 5% trimmed 
mean (Figure 3.14) seems to perform as well as the bootstraped sample mean; it is 
better for very small samples, say for n=5, 10, and 15. This confirms the general 
relationship among these estimators, even in the case of bootstraping the estimators, 
that the 5% trimmed mean is a robust compromise between the sample mean and the 
sample median. 

The results obtained in this experiment, however, do not agree with the classical 
theory in the case of the Laplace population. In this case the bootstraped sample mean 
outperforms the bootstraped sample median in estimating the center of the 
distribution, for sample size nS 20. For a sample of size n = 60, there is no real 
difference between these two estimators, in terms of MSEs". Notice that the 5% 
trimmed mean (Figure 3.17) performs better than the bootstraped sample median 
(Figure 3.16) for the cases where n<60, but in turn, is outperformed by the 


bootstraped sample mean ( Figure 3.15). 


E. LINEAR REGRESSION BY BOOTSTRAPING THE RESIDUALS 

In a final experiment, linear regression estimation was considered. In this case, 
there is a choice of bootstraping methods; however, in this thesis only one method 1s 
considered. The method considered here relies on bootstraping residuals to estimate 
the variance of the B" vector(B” stands for ” B hat”. A measure to estimate the MSE 
of this vector is also introduced. 

In the typical linear regression problem there are n independent observations 


(real-valued) Y: and it is assumed that the following model holds: 

Y= xb ΤΕ. (3.16) 
where € is a random sample from some population F, and j isa p X 1 vector of 
unknown parameters that must be estimated. All that is assumed about F is that it ıs 


centered at zero, E(e)= O and Cov(z)2- o* I . One way of estimating f is by the 


commonly used least squares method, in which the sum of the squared distances 
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between the y; and the predicted values vg is minimized. When this fitting technique 1s 


used, the estimate of B® is obtained by choosing the B^ such that: 


bb: min AG ye Ст 
р 


Then, as ıs well known 
bh = (X XYIXY , (3.18) 


where X’ stands for the transpose of X . Also, the vector € can be estimated by the 
vector of residuals, 


gh 


= у; -у;'. (3.19) 
[t is desired to determine the precision of the estimator B” . The bootstrap method 
could now be applied to estimate the variability and the MSE of the vector f by 
bootstraping the residuals. [ As a remark, the second method discussed by Efron 
[Refs. 1,4: Section 7.2,7], considers each covariate response pair Z; = (Y; , X; ) to be a 
single data point obtained in the p X 1 space by sampling from Е“ randomly. 
Therefore, this method does not condition on X and does not presuppose that the 
model (3.16) in question is correct. It estimates the joint distribution of Y and X;. 
Then, the algorithm presented in Chapter Two could be used to estimate the 
covariance matrix of PP]. 
The algorithm for bootstraping explained in Chapter Two, Section A.2, can be 

used as follows: 

(1) construct FP , by giving mass 1/n at each observed residual and sample F^ to 
obtain bootstrap samples: e: ~ gF : 


(2) construct a new data ү” call this the bootstrap data set, by using t. and p^ 


* 


У = ХЕ. (3.20) 


ο 


(3) Using the same fitting technique used to obtain B" in the original problem, 
* . р * 
calculate P . Then obtain an estimate ofß : 


b = (XX)1X Y Bo) 


(4) Repeat steps (2) and (3) B times obtaining independent bootstrap 
realizations Ж би δ ῃ . Then the covariance of B^ can be estimated by 
the sample covariance matrix of the b^, p=" B. 
Efron has shown (See [Ref. 1: page 18] ) that as B - © , 


Var(f ) = ((n-p)/n) (X' Xy? (3.22) 


where 6? is an unbiased estimate of the variance of Y; . In this procedure, 0? can be 


estimated by Eo: . It can be seen that as B — 99, 
Var(ß ) > Var(ß®). (3.23) 


The following experiment was conducted to estimate the MSE of B®. Suppose it 
is known that the observations Y: come from a Normal(0,1). Then the true value of the 
D- vector in the regression model (3.17) is D = (0,0,0), ѕо the E(B) = O and the 
variance-covariance matrix of B is Lp = c*(X' Xy1, where itis known that o* = 1. 

For this experiment, a design matrix X of orthogonal-column vectors was 
created. This matrix has l's in the first column; then a series of n alternating l's and 
-]'s in the second column; and finally the third column (for p= 3 ) is a series of two 1's 
and two -1's (also, n = 2% ,x = 2, 3, 4,... ). Then it was possible to readily calculate 
p^ , by 


ODIA (3.24) 


The bootstrap algorithm described above was used to generate a sample of В; len: 


an estimate of D: is 


Б. = (I/n (X' Y). (3.25) 
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It was desired to develop a measure of precision for В” analogous to MSE, which 
depends on Var(ß ) and the bias of p . Define 


MSE(B”) - EK" - E»). (3.26) 


Recall that in this experiment the E(B" ) = O. Then, (3.26) could be estimated in the 
following way: 


1) Do step (4), as above, obtaining 


* * . 
М5Е«(В ) = [Mii - ED^»)*]/B . i9 L2. B (3.27) 
ж 
= >B, -BH ]/B . 
2) Repeat (1) a number of M times to obtain an average MSEx" of the procedure 
(3.27). 


The results of this experiment are shown in Figure 3.18. 
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Figure 3.18 Estimated Averages MSE of p^. 
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Here, the sample sizes were taken as n = 4, 8, 16, 32, 64, and 128, and M = 15. 
The estimator B was bootstraped a number B = 5, 10, 15, 20, 30, 40, 50, 100, 150, 
and 500. The results obtained were surprising. When the number of observations is 
small, n<33 , the MSEx" of the estimator is relatively high (MSE«^ > .09) even 
when B is as large as 500. When n > 65, there is some improvement in the MSE«P ; in 
this case, the MSE" is at least 5% lower that when ton < 33. It is interesting to see 
that increasing B from 5 to 500 there is no remarkable gain in the precision of 
estimator when n > 65; the MSE:® oscillates around the same value. Now, when n < 
33, increasing B by the same amount, the MSEx" decreases but less than 1% of its 
initial value. It seems that in the linear regression estimation the key problem is the 
size of n and not of B. 

When using this method for estimating the MSE of B" , the practitioner must 
bear in mind that it involves the residual distribution and hence assumes that the linear 


model is correct. 
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IV. CONCLUSIONS 


As it has been shown, the Bootstrap is an accurate method for esumating the 
precision of the estimates and for estimating the distribution (or some feature of the 
distribution) of an estimator. For MSE, the number B required to obtain a certain 
degree of accuracy will vary depending mainly on the population (this is a subject for 
further studies) and the type of the estimator used for estimation. It was found that 
when the sample comes from a population having heavy-long tails, such as the Laplace 
distribution, the bootstrap estimator for the mean is a better estimator for estimating 
the center of the distribution than the median or the 5% trimmed mean; where in the 
case of using nonbootstrap estimators, the median is a better estimator than the other 
two estimators. 

In estimating the variance of a population, it was found that there exists an 
estimator that is more accurate than the typical estimator recommended in the 
bootstrap literature. This estimator (5 ®) relies on the fact that the original sample 
mean in the bootstrap method is known. Once this value is calculated, there 1s no need 
to find X for each bootstrap sample, since X is fixed through the process. Another 
estimator for c? was also proposed, E . This estimator 1s unbiased, where 5 1$ 
not, but for small sample sizes, n « 30 , is not as accurate as s It should be 


emphasized that in using this estimator, „S °, one can reduce the computer time 


' 3 
required to estimate 6%. Hence, this is another advantage in using this estimator. 

In the linear regression estimation, using as a measure of precision definition 
(3.28), it was found that the bootstrap method analyzed in this thesis gives estimates 
with small MSE" with relative small sizes of В, but for relatively large sample size, п 
> 60. When the sample size is small, increasing B up to 500 will result in a gain of 
around 1% in the precision of the estimates. Thus, in the linear regression estimation 
the critical issue for MSE is the sample size. It was also noted that the disadvantage of 
this method is that it assumes that the model in question is correct. 

The result that seems to apply to all cases studied in this work is that, in using 
the bootstrap method for estimating MSE of some parameter 0 , there really exits a 
tradeoff between B and n: as n increases, one can significantly decrease B and still get 


very precise estimates. However, no matter what n is, once some degree of accuracy 


4] 


has been obtained, there is no reason to increase B much more since this will not 
induce greater precision in the estimates. In Appendix C , the reader will find tables 
that provide information about this tradeoff for given estimators and populations. 
Analyzing the figures presented in previous chapters and these tables, a rule of thumb 
about the relation between n and B can be hypothesized. The following rule seems 
reasonable: make the number B ^ 1000/n. In almost all cases studied here, this rule 
yielded estimates with. MSE«^ < 0.05 (note: independent of n, making 40 < B > 
60 will also produces estimates with small MSE«^ ). The only exception is when the 
population in question was Laplace(0,1). This is an area that needs further study. 

Finall, it was found that a (possibly not serious) disadvantage in using the 
bootstrap method is the computer time required to obtain the estimates. For example, 
in estimating the variance of a Gamma(0.5,1) distribution, increasing B from 20 to 100 
increased the CPU time of the IBM 3033-A16 system used in this experiment about 
75%. This time is increased at least another 50% if one desires to obtain the 
distributional characteristics of the estimator (1.e., boxplots). However, in view of the 
decreasing cost of computer time, this does not seem to be a major obstacle for using 
this method. 
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APPENDIX A 
LIST OF SPECIAL NOTATIONS 


(1202 ; -hat, estimator of 0 

(2) F^ :empirical probability distribution 

(3) ӨЕ") :the value of 0 based on bootstrap method 
(4) x" ‚а bootstrap random sample 

(5) MSE," :estimated MSE based on bootstrap method 
(6) p^ ‘estimator of the p X 1 p -vector 

(7) b^ :an estimate of Bh 

(8) p ‘estimator of B based on bootstrap method 
(9) b :an estimate of p 
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APPENDIX B 
FORTRAN CODE FOR BOOTSTRAPING 


This program, called BOOTST, was developed to estimate distributional 
properties of some statistical estimators using the Bootstrap Method. Also it is possible 
to obtain estimates of the MSE of the estimators. The code was written in FORTRAN 
77. It can generate a random sample for Monte Carlo simulation or can read the 
sample data by a CALL to a subroutine FDATA (at the end of the code listed below). 
The user can generate samples from the following distributions: Exponential(A), 
Laplace(0,1), Uniform(0,1), Normal(0,1), Gamma(a,l), Poisson(A), and the 
Geometric(p). The parameters d, À, and p can be specified by the user within the 
appropriate function. With this program, the user can study the distributional 
properties of the following bootstrap estimators: mean, variance coefficient of 
variation, serial correlation, median, and the 5%-trimmed mean. Also, one can obtain 
estimates of the “B -vector” in the case of the linear regression estimation by 
bootstraping the residuals ( See Chapter Three, Section D ). The program is structured 
in five main sections: the MAIN program, to include input requirements; the DATA 
GENERATION, the ESTIMATORS definition, the BOOTSTRAP SAMPLING 
mechanism, and the STATISTICS sections. 

The program can be used in two ways. The first, makes use of another program 
called SMTBIO. This code was developed at the NPGS by Prof. P.A.W. Lewis, and 
Mr. Luis Uribe (See (Ref. 9] ). It is highly recommended that the user become familiar 
with the documentation of STMBIO before attempting to use BOOTST. In general, 
when using this option, the user must create an input file containing the parameters 
specified in the input section of BOOTST. Then, a CALL is made to STMBIO, and in 
turn STMB10 will make various sequential calls to generate the data, calculate the 
values of the desire estimators (using the bootstrap mechanism), and produce the 
statistics. When a call to STMBIO is made, the user could produce estimates for 1, 2, 
or 3 different estimators using 1, 2, or 3 sample data generators or any of the eight 
possible combinations. Also, the user could select up to 8 different sample sizes for 
each estimator. Therefore, in one execution, statistics for up to three different 


estimators, using up to three different data generators, and for up to eight different 
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sample sizes can be obtained using the bootstrap method. These options are controlled 
in the INPUT requirements of BOOTST. At the end of each execution, BOOTST will 
send to a printer (or to the screen, depending on the option selected) a file containing 
boxplots and a summary of the statistics for each estimator. The input requirements 
are controlled by the user in a file called BOSIN. 

The general execution of BOOTST runs as follows: 


(1) For each estimator 


(2) Read Input Requirements (MAIN) 

(3. CALL STMBIO 

(4) CALL Data Generator (Data Generation Section) 
(5) N=kX n random variates are generated, where k= 1 or 2,..., 


or 8 different sample sizes. Then the data is sectioned into 
samples of sizes N(K)=n. If M repetitions of the process are 
allowed, then a total of MX N random numbers are obtained. 
Estimates are calculated for each sample size N(K). 


(6) CALL Estimator Function | ( Estimator Section) 
Begin Generation of Estimates 
(7) For I= ] to B | 
CALL BOOTSTRAP (Bootstrap Section) 


CALL STATISTIC 
Store Bootstrap Estimates 
CALL STATI TIC | 
Store Mean of Bootstrap Estimates 
(8) PRODUCE Boxplot and Statistics 
The input requirements specific to BOOTST are explained below, the other 
inputs declared in the MAIN are specific to STMBIO ( See [Ref. &ref10] ). 


(1) ANS : 10r O : If the user wants to, store each bootstrap estimate for each 
estimator, the answer should be 1. Estimates are stored 1n FILE 21. 


(2)  NE(I) a vector containing the sample sizes (n). Up to 8 different sample sizes. 
(3) IB: Number of bootstrap replications for each execution. 
(4) IX: Seeds used to generate data (up to 3 different seeds). 

If the user desires to obtain estimates and graphical displays of two or more 
different estimators and ıs using a large number B, say B 2 60, the amount of 
computer time required will increase significantly depending on the system used. 

The second way to execute BOOTST is recommended for more experienced users 
or for those who do not want to obtain boxplots ofthe estimates. This option will save 
a great deal of CPU time. For this option, the user will have to make some simple 
changes to the MAIN program: 


] Delete from the input requirement section those inputs that only apply to 
w STMB10 (those not listed л i di» 


(2) Replace the call to STMBIO by the following sequence of calls: 


45 


i) Call Data Generator (1.e., one of the data generators) | 
11) Call Estimator (ie., one of the estimator functions) The estimator 


function (subroutine) will make the appropriate call to thé Bootstrap and 
Statistic subroutines. 


(3) For this option, the input parameters ANS must be set to integer 1. Also, if 
the user, now make reference to the code, it will be noticed that each estimator 
subroutine has a special parameter WI. This parameter must be deleted 
everywhere since its only applies to STMBIO. 


The computer code 1s listed below. 
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APPENDIX C 
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Figure C.4 Bootstrap Dist. of Sample Mean В= 150. 
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